Remote gaming applications, in which a server-side game is controlled by a client-side player, have attempted to encode the video output from a three-dimensional (3D) graphics engine in real-time using existing or customized encoders. However, the interactive nature of video games, particularly the player feedback loop between video output and player input, makes game video streaming much more sensitive to latency than traditional video streaming. Existing video coding methods can trade computational power, and little else, for reductions in encoding time. New methods for integrating the encoding process into the video rendering process can provide significant reductions in encoding time while also reducing computational power, improving the quality of the encoded video, and retaining the original bitstream data format to preserve interoperability of existing hardware devices.
Typical video rendering pipelines are separate and independent from video encoding pipelines, with little crossover between process and expertise in the two domains. As a result, some of the visual effects and post-processes applied in the video rendering pipeline are counterproductive to the video encoding process, leading to video artifacting, increased encoded video size, and longer encoding times. However, these visual effects are still desirable in the resultant decoded video.
By integrating video rendering and video encoding pipelines, post-process effects can be deferred to improve the encoding process. For example, simulated filmic grain introduces randomly-occurring animated grain that is difficult for typical encoders to process without a significant cost to video quality or compression ratio. Some video encoding methods attempt to remove this additional visual noise before encoding, but these methods are offline-only and computationally costly. By disabling this specific post-process in the rendering pipeline, the video automatically becomes easier to encode. The post-process can then be applied after the video is decoded. In the case of filmic grain, compositing the grain over the decoded video is not computationally demanding, can be done in real-time at the decoder, and may improve subjective video quality by disguising other encoding artifacts.
International Patent Application No. WO2016172314 A1 (“the 314 Application”) discloses systems and methods directed to artistic intent based content coding. A coding user interface permits a user to specify an artistic set and to configure treatment of pixels and/or blocks associated with an artistic set, such as a fidelity enhancement, QP adjustment value and/or post-processing. Examples of artistic intent that may be added to video output include when an encoder may remove the film grain from the original signal before encoding and use the film grain SEI to convey to the decoder how to regenerate the film grain and add it back to the video signal before it is displayed. The present invention may be distinguished from the '314 Application at least because the '314 Application does not disclose disabling specific post-processes in the rendering pipeline prior to encoding and then applying those post-processes after the video is decoded. As a consequence, the present invention is an improvement to this computer technology because it offers improved encoding and decoding of video data without a significant cost to video quality or compression ratio. The present invention is also an improvement because it improves the resulting bandwidth, bitrate, encoding time, and is capable of being used in real-time video streaming applications with improved video quality.
U.S. Pat. No. 9,609,330 (“the '330 Patent”) discloses content adaptive entropy coding of modes and reference types data, meaning that a pre-analyzer subsystem of the encoder analyzes content to compute various types of parameters useful for improving video coding efficiency and speed performance. These parameters include horizontal and vertical gradient information (Rs, Cs), variance, spatial complexity per picture, temporal complexity per picture, scene change detection, motion range estimation, gain detection, prediction distance estimation, number of objects estimation, region boundary detection, spatial complexity map computation, focus estimation, and film grain estimation. The parameters generated by the pre-analyzer subsystem can then be consumed by the encoder or be quantized and communicated to the decoder. The present invention may again be distinguished from the technology disclosed in the '330 Patent at least because that technology does not disable specific post-processes in the rendering pipeline prior to encoding and then apply those post-processes after the video is decoded. The present invention is therefore an improvement to the computer technology of the '330 Patent because it offers improved encoding and decoding of video data without significant a cost to video quality or compression ratio and because it is capable of being used in real-time video streaming applications with improved video quality.
U.S. Pat. No. 9,762,911 (“the '911 Patent”), discloses systems and methods for techniques related to content adaptive prediction and entropy coding of motion vectors. The technology disclosed allows for a first video data and second video data to be received for entropy encoding at an entropy encoder module. The first video data and the second video data may be different data types (e.g., header data, morphing parameters, synthesizing parameters, or global maps data or motion vectors or intra-prediction partition data or so on, as is discussed further herein). A first entropy encoding technique may be determined for the first video data based on a parameter associated with the first video data such as, for example, a number of compressed bits of the first video data, a predetermined indicator or flag associated with the first video data, a predetermined threshold, or a heuristically determined threshold or the like. In some examples, the first entropy encoding technique may be chosen from one of an adaptive symbol-run variable length coding technique or an adaptive proxy variable length coding technique. The first video data may be entropy encoded using the first entropy encoding technique and the second video data may be entropy encoded using the first entropy encoding technique. Once more, the present invention is distinguishable at least because the technology disclosed in the '911 Patent does not involve the selective disabling of post-processes in the rendering pipeline prior to encoding and then apply those post-processes after the video is decoded. Once again, the present invention is an improvement to the computer technology of the '911 Patent because it offers improved encoding and decoding of video data without a significant cost to video quality or compression ratio. The present invention is also an improvement because it improves the resulting bitrate, encoding time, and is capable of being used in real-time video streaming applications with improved video quality.
As is apparent from the above discussion of the state of art in this technology, there is a need in the art for an improvement to the present computer technology related to video encoding in game environments.